Interactive and Incremental Learning Via

As computers are widely used and computer-programming gets increasingly complicated, computer users and programmers demand more convenient human-computer interfaces and programming tools. Motivated by facilitating computer programming and human-computer interaction, this project explores teaching a computer to react properly to external stimuli through natural human-computer interaction. The long-term goal of the project is to program a computer as we teach an infant, and to enable the computer to interact with us like a human and perform jobs accordingly. The project uses a multisensory mobile robot as the interface for natural human-computer interaction. We develop a computational efficient scheme that facilitates the robot to learn spoken language online, and react properly to learned speech commands. Compared to the existing speech recognizers, our system does not use text or other symbolic information to represent speech, thus is not restricted by the limitations that a speech-to-text mechanism may inherently have. Due to this fact, our system can learn speech online in any language. This learning flexibility is not achieved by state-of-the-art speech recognizers. The thesis reports the design and implementation of our scheme for spoken language acquisition. In the thesis, we discuss the available resources and possible pitfalls for the robot project; we discuss the speech communication techniques that can be or cannot be used in our project; we choose the classifiers that are suitable to our project; and we also present a human-computer interaction model and the robot training strategies. Based on these analyses and discussions, we develop our learning scheme that can learn online from both labeled and unlabeled data. The learning scheme is based on successive refinement of the decision boundary. It learns incrementally through its interaction with the teacher. Experimental results of a multilingual short-utterance learning task are given after the presentation of the learning scheme. In the learning experiment, both speech and tactile sense are incorporated. Further extension of this project includes incorporating more versatile modalities into human-computer interaction, and applying our learning scheme to practical systems such as home appliances. iv Acknowledgements